Fix HDF5 dataset creation for large non-scalar variables #659
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dqwu
added
enhancement
High Priority
Next Release
For HDF5 type, record variables are currently chunked using the full dimension length. With very large dimensions, this can cause dataset creation to fail in the HDF5 library. This test simulates an output history file from the E3SM G case with extremely large dimension lengths to reproduce the issue. It will also help guide the upcoming fix for the default chunking strategy used by HDF5 type.
For HDF5 type, record variables are currently chunked using the full dimension length. With very large dimensions, this can cause dataset creation to fail in the HDF5 library. Introduce a configure option to set the chunk size (in bytes) for HDF5/PnetCDF chunked variables, with a default of 4 MB.
For HDF5 type, all record variables were previously chunked using the full dimension length. With very large dimensions, this could cause dataset creation to fail in the HDF5 library. Introduce a simple, effective chunking strategy inspired by NetCDF4, which now applies to all non-record (non-scalar) variables as well. This ensures reliable dataset creation even for large dimensions.
jayeshkrishna
force-pushed
the
dqwu/fix_hdf5_chunking
branch
from
574969c to
1a709f6
Compare
Refactoring function that defines an HDF5 variable. * Moving from C to C++ data structures * Splitting the long function into multiple functions to make the code easier to read * Refactoring/rewriting code and adding more comments to make the intent of the code clearer * Fixed minor leaks on error cases when creating the string type
Print the HDF5 error stack on error if HDF5 is enabled
Moving cmake option to close+open to sync test output from a CMake dependent option to a regular option. This change allows us to explicitly enable this option as needed during configure. The option is turned on by default for ADIOS.
Enabling close+open to sync output data for HDF5 tests This is required since we currently use NetCDF4 to read the data written out using HDF5
Instead of returning after printing the usage we now add the option into the list of "no val" options. This allows user to query and handle the option accordingly
Generalizing the test code in test_hdf5.c and test_adios.c so that it can be used for all I/O types. This commit adds the boiler plate for this executable. We now have, * Command line arg handling (to specify I/O types, rearrangers, number of I/O tasks etc) * A bunch of util functions We do not have the F/G/I test code yet, however we have the boiler plate functions that will be modified to add this feature
A generic framework to easily define and write variables is now added to the C++ E3SM example. The framework contains abstractions for, * I/O decompositions * Variable dimensions, attributes * Different kinds of variables (variables to write with starts/counts, variable with and without unlimited dimensions) The test code for F/G/I cases were also updated to add dummy variables similar to the ones written out by E3SM F/G/I cases. (Also see examples/c/test_hdf5.c to see older version of these examples) Functionality to verify the contents written out will be added in future
Adding source code comments. No code change
jayeshkrishna
force-pushed
the
dqwu/fix_hdf5_chunking
branch
from
93234b4 to
591d151
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Large HDF5 record variables previously failed during dataset
creation because chunking used the full dimension length.
This PR applies a simple, effective chunking strategy (based
on NetCDF4) to all non-scalar variables, adds a configurable
chunk size for HDF5/PnetCDF, and includes a unit test that
reproduces the issue for large-dimension variables.
These changes ensure reliable dataset creation for very large
variables and improve robustness for HDF5 while preparing for
future PnetCDF chunking support.
This PR also includes code refactoring of the function that
defines HDF5 variables.
A new C++ example (based on test_hdf5.c) is added and will
be used later to replace test_[hdf5 | adios].c examples.